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1.  Introduction 


In  recent  years  statisticians  have  become,  increasingly  concerned  with  the 
meaningful  formulation  and  solution  of  certain  multiple-decision  problems  which 
arise  in  experimentation.  Thus,  for  example,  when  an  experimenter  conducts  tests 
to  compare  the  performances  of  several  competing  categories  of  items,  his  ulti- 
mate objective  often  is  to  select  the  category  (or  categories)  which  is  (are) 
best,  goodness  being  measured  in  terms  of  a particular  parameter  (e.g.,  the 
population  mean  or  the  population  variance)  associated  with  the  random  variable 
being  observed.  To  accomplish  this  the  experimenter  requires  a statistical 
decision  procedure  which  will  tell  him  how  many  observations  to  take,  how  to 
take  these  bbseryations,  and  based  on  these  observations  which  population^ ) to 
choose;  the  decision  procedure  should  have  the  property  that  the  probability  of 
an  incorrect  selection  (or.  more  generally,  the  risk  or  expected  loss)  is  con- 
trolled at  some  specified  level. 

In  response  to  the  need  for  such- decision  procedures,  research  statisti- 
cians have  been  studying  various  possible  appropriate  formulations  of  these 
problems,  and  have  developed  a body  of  statistical  methodology  to  cope  with 
them.  The  procedures  have  come  to  be  referred  to  as  ranking  and  selection 
procedures,.  The  purpose  of  this  paper  is  to  introduce  the  reader  to  these 
procedures,  to  describe  some  of  them  and  the  philosophy  underlying  their  use, 

i 

and  to  discuss  their  properties.  % 

In  Section  2 we  will  pose  th4  normal  means  problem,  and  use  it  as  a 
vehicle  for  motivating  some  of  the  bspic  ideas.  The  two  most  commonly  adopted 
formulations  of  ranking  and  selection  problems,  namely  the  so-called 
indifference-zone  approach  and  the  subset  approach,  will  be  described.  The 
attributes  of  single-stage,  two-stage.,  and  sequential  procedures  devised  for 
the  normal  moans  problem,  under  different  assumptions  concerning  the  population 
variances,  will  be  assessed.  In  Section  3 we  sketch  some  analogous  results  for 
the' normal  variances  problem,  and  in  Section  4 we  mention  results  for  parameters 
of  other  distributions. 


This  research  was  supported  in  part  by  the  U.S.  Army  Research  Office- 
Durham  under  Contract  DAIIC04-73-C-0008  and  by  the  Office  of  Naval  Research 
under  Contract  N00014-67-A-0077-0020. 
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The  number  of  research  papers  written,  on  subjects  in  this  field  is  now 
vast;  it  / is  hoped  that  this^rief  introduction  will  stimulate  the  reader  to 
explore  the  literature , and  to  apply  the  procedure's  where  appropriate . 

2.  The  normal  means  problem 

A very  important  problem- which  arises  frequently  in  applications  is  that 
of  selecting  the  normal  population  which  has  the  largest  population  mean.  Thus, 
for  example,  'the  ordnance  engineer  might  be  conducting  firing  programs  to  com- 

i * , ' , 

pare  the  ballistic  performance  of  different  types  of  projectiles  (in  which  case 
his  objective  might  be  to  select  that  type  which,  on  the  average,  travels  the 
greatest  distance),  or  the  medical  research  worker  might  be  studying  the  response 
of  patientc  to  different  kinds  of  analgesic  drugs  (in  which  case  his  interest 
might  lie  in  selecting  that  drug  which  produces,  on  the  average,  the  longest 
period  of  time  without  pain),  or  the  agronomist  might  be  conducting  field  trials  • 
with  different  varieties  of  grain  (in  which  case  his  purpose  might.be  to  selept 
thst  variety  which  produces,  on  the.  average,  the  largest  yield  per  acre).  In  all 
of  these  cases  large  values  of  the  means  are  deemed  to  be  desirable;  however,  in 

• . i 

other  cases  small  values  of  the  means  might  be  considered  desirable.  The  proce- 
dures that  we  will  describe  can,  with  minor  modifications,  handle  these  latter 

; i 

cases  as  well. 

i t ' . 

In  Sections  2.1  and  2.&.1  we  shall  state  the  statistical  assumptions  , 
which  underlie  the  procedures  that  have  been  developed.  Then  we  shall  describe 
several  approaches  to  the  selection  problem. 


2.1  Statistical  assumptions  - 
* 

We  shall  assume  that  we  have 


k sources  H.  (l£,i,£jO  of  normally 


distributed  data,  the  ith  source  having  population  mean 

* 


u.  and  population 


variance 


population  (l£i£)<)  should  be  thought  of  as  being  associated 

with  the  ith,  categbry.  The  p*  are  assumed  to  be  unknown.  Let  Pro  «.  V---  < 


no 


n 

denote  the  ranked  values  of  the  p 


i’ 


[1]-hC2]« 
it  is  assumed  that  the  pairing 


of  the  nL  with  the  p^  3,1)0  -is  completely  unknown.  Possible  assump- 

tions concerning  the  Cj  Tl<,i<)<)  .will  bo  discussed,  in  Section  2.3.1.  Throughout 
this  paper  X,^  (X^i<j< , j—1  *2 , . . . ) will  denote  the  jth  observation  from  11^, 

all  observations  being  assumed  independent.  , 
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2.2  Soma  formulations 

The  two  moat  commonly  used  formulations  of'  the  selection  problem  are 
due  to  Bechhofer  [1954]  and  Gupta  [1956],  [1965];  these  are  referred  to  as  the 
Indifforence-zone  approach  and  the  subset  approach,  respectively,  The  approaches 

are  described  below. 

: - 2.2.1  The  indifference- gone  approach 

The  goal  and  probability  requirement  associated  with  the 
Indifference-zone  approach  are: 


Goal:  "To  select  the  population  associated  with 


(2.1) 


It  is  assumed  that  prior  to  the  start  of  experimentation  the  experimenter  can 
' specify  two  constants  (Q<fi*<®,l/k<P*<l)  which  are  then  incorporated 

into  the  following  probability  requirement: 


Trobabillty  requirement:  . 

Frob{Selecting  the  population  associated  with  yr,  > P* 

LkJ  (2.2) 

Whenever  n[1(]  - > 8*. 


The  experimenter  then  restricts  consideration  to  procedures  which  guarantee  (2.2). 
(In  (2.2)  the  specified  quantity  5*  can  bo  thought  of  as  the  smallest  difference 
"worth  detecting"  between  the  population  mean  of  the  "best"  and  "second  best" 
population;  P*  is  specified  strictly  greater  than  1/k  since  a probability  of 
1/k  can  be  achieved  by  choosing  one  of  the  k populations  at  random. ) 


2.2.2  The  subset  approach 

The  goal  and  probability  requirement  associated  with  the  subset 

approaoh  are: 

Goal:  "To  select  a (non-empty)  subset  of  the  populations  (23) 

which  contains  the  population  associated  with  Uj.^." 

It  is  assumed  that  prior  to  the  start  of  experimentation  the  experimenter  can 
specify  a constant  (P*)  (l/k<PA<l)  which  is  then  incorporated  into  the  follow- 
ing probability  requirement; 
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'Probability  requirement : 


Prob{Selected  subset' contains  the  population  associated  with 
regardless  of  the  values  of  the  ( l<,i  <,k ) . 


'tk]}  i p* 

(2.4) 


The  experimenter  then  restricts  consideration  to  procedures  which  guarantee  (2.4). 

i 

Remark  1:  It  is  to  be  noted  that  the  experimenter  plans  his  experiment  assuming 

that  the  population  fneans  are  n6t  all  equal;  this  is  a very  reasonable  assumption 
in  almost  all  real-life  situations.  He  is  interested  in  identifying  the  "best" 
population  — in  this  case  the  population  with  the  largest  population  mean.  Goal 
(2.1)  leads  to  a k-declsion  problem  since  the  experimenter  must  choose  one  of  the 
k populations  based  on  the  outcome  of  his  experiment  (i.e.  , his  possible  deci- 
sions are:  is  best,  or  is  best,  or  * • • , or  is  best).  Similarly, 

goal  (2.3)  leads  to  a (2^-1 )-decision  problem  since  the  experimenter  must  choose 
one  of  the  2 -1  non-empty  subsets  of  the  k populations  based  on  the  outcome 
of  his  experiment,  (e.g. , for  k=3  his  possible  decisions  are:  only  is  in 

the  subset,  only  is  in  the  subset,  only  P.3  is  in  the  subset,  and  Il2 

are  in  the  subset,  11^  and  Il3  are  in  the  subset,  Il2  and  II3  are  in  the 
subset,  and  1I2  and  IIg  are  in  the  subset).  These  multi-decision  approaches 
are  in  marked  contrast  to  the  classical  2-decision  test-of -homogeneity  approach 
afforded  by  the  Analysis  of  Variance;  in  that  approach  the  experimenter  tests  the 
(usually  completely  unrealistic)  hypothesis  that  the  . k population  means  are 
equal,  and  decides  based  on  the  outcome  of  the  experiment  either  to  accept  the 
hypothesis  or  to  reject  the  hypothesis. 


Remark  2:  As  noted  above,  goal  (2,1)  leads  to  a k-decision  problem.  However, 

depending  on  the  practical  situation  under  consideration,  the  experimenter  can, 
using  the  indifference-zone  approach,  pose  more  general  goals.  For  example,  he 
may  wish  to  select  the  t (I<,t<)<-1)  best  populations  with  regard  to  order,  or 
he  may  wish  to  select  the  t (l<t<)c-l)  best  populations  without  regard  to  order, 
t being  fixed  before  the  start  of  experimentation.  (Both  goals  reduce  to  (2,1) 
when  t*l. ) These  more  general  goals  lead  to  a [k!/(k-t)J ]-decision  problem 
and  a [k!/tl (k-t )l  !l-decision  problem,,  respectively . Such  general  goals  and 
others  are  discussed  in  Bechhofer  [1954]  and  Mahamunulu  [1967], 
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Remark  3:  For  goal  (2.1)  and  the  indifference-zone  approach,  the  experimenter 

always  ends  up  by  selecting  a single  population.  For  goal  (2.3)  and  the  subset 
approach,  the  experimenter  ends  up  by  selecting  1*  or  2 or  •••  or  k 
populations,  depending  on  the  outcome  of  the  experiment;  thus  for  this  latter 
approach  the  number  of  populations  in  the  selected  subset  is  a random  variable. 

2.2.3  Other  approaches 

Santner  [19753  has. proposed  a restricted  subset  approach  in  which 
the  experimentor  selects  1 or  2 or  •*•  or  c populations,  depending  on 
the  outcome  of  the  experiment,  where  c (l<,c<JO  is  decided  on  and  fixed  before 
the  start  of  experimentation;  his  approach  can  be  regarded  as  bridging  the 
indifference-zone  and  subset  approaches  since  if  c=l  his  approach  reduoe3  to 
the  indifference-zone  approach  while  if  c=k  it  reduces  to  the  subset  approach. 
Other*  approaches  in  which  more  general  "loss  functions"  are  used  have  been  pro- 
posed by  Somerville  [1954]  and  Fairweathcr  [1968].  An  approach  in  which  the 
are  assumed  to  have  prior  distributions  has  been  considered  by  Dunnott  [I960] 
while  a similar  idea  from  a Bayesian  point  of  viow  has-been  proposed  by  Roiffa 
and  Schlaiffer  [1961]  and  Deely  and  Gupta  [1968].  However,  for  brevity  we  will 
not  discuss  these  or  other  approaches. 


2.3  Assumptions  concerning  the  variances 


2.3.1  Possible  assumptions 

In  order  to  devise  procedures  which, will  guarantee  (2.2)  or 

(2,4)  for  the  normal  means  problem,  it  is  necessary  to  make  an  assumption  con- 
* 2 

corning  the  values  of  the  (l^A^)-  Which  assumption  it  is  appropriate  for 

the  experimenter  to  make  in  any  particular  practical  situation  depends  on  tho 

information  available  to  him  at  tho  time  that  he  plans  his. experiment.  The  four 

moat  common  assumptions  are  that; 

2 

a)  The  values  of  the  c, 

2 * 

equal  to  o (say). 


(l<i<k)  are  known,  and  all  are 

(2.5a) 


b) 


2 

Tho  values  of  the 
are  equal. 


(l,<i<k)  are  known.,  but  not  all 


(2.5b) 


c)  The  values  of  the  o?  (l<,i<V)  are  unknown,  but  it  is 

1 -2 

known  that  they  have  a common  value  o (say). 

2 

d)  The  values  of  tho  (l£i<k ) are  completely  unknown. 


(2.5c)  ' 
(2. 5d) 
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2.3.2  The  variance  assumption  and  associated  procedure? 

Once  the  experimenter  has  adopted  one  Of  these  assumptions  he  then 
must  choose  a selection  procedure  which  was  derived  under  that  particular  assump- 
tion. 

Thus,  for  example,  if  he  wishes  to  guarantee  (2.2)  and  adopts 
assumption  (2.5a)  or  (2.5b),  then  he  can  use  a single-stage  procedure  (Bechhofer 
[1954]),  a two-stage  procedure  ,(Alam  [1970]  or  (Tamhane  [1975]),  an  open  sequen- 
tial procedure  without  elimination  (Bechhofer,  Kiefer,  Sobel  [1968]),  or  a closed 
sequential  procedure  with  elimination  (Paulson  [1964]).  If  he  wishes  to  guarantee 
(2.2)  and  adopts  assumption  (2.5c),  then  he  cannot  use  a single-stage  procedure 
(see  Dudewicz  [1971])  although  he  can  use  a two-stage  procedure  (Bechhofer, 
Dunnett,  Sobel  [1954])  or  a sequential  procedure  (Paulson  [1964]);  similarly, 
if  he  wishes  to  guarantee  (2.2)  and  adopts  assumption  (2.5d),  then  he  cannot  use 
a single-stage  procedure  although  he  can  use  a two-stage  procedure  (Dudewicz  and 
Dalai  [1971]  or  Rinott  [1974]).  Finally,  if  the  experimenter  wishes  to  guarantee 
(2,4),  and  he  adopts  assumption  (2.5a)  or  (2.5c),  then  he  can  use  a single-stage 
procedure  (Gupta  [1956],  [1965]). 

When  the  experimenter  has  adopted  a particular  assumption  and  as  a 
consequence  has  the  option  of  choosing  among  several  competing  procedures,  each 
one  of  which  will  guarantee  his  probability  requirement,  he  then  chooses  one  of 
these  procedures  on  the  basis  of  various  possible  operational  or  cost  criteria. 

An  indication  of  such  criteria  will  be  given  in  our  later  discussion.  In  the 
next  section  we  shall  describe  certain  selection  procedures.  Our  emphasis  will 
be  on  procedures  which  can  be  used  with  the  indifference-zone  approach  to 
guarantee  (2.2). 

2 . 4 Procedures  for  use  with  the  indifference-zone  approach  under  the 

assumption  of  common  known  variance 

In  this  section  we  shall  describe  three  procedures,  each  one  of  which 

will  guarantee  (2.2)  when  assumption  (2.5a)  is  made;  minor  modifications  of  these 

procedures  will  guarantee  (2.2)  when  assumption  (2.5b)  is  made.  The  procedures 

will  be  introduced  in  the  order  of  their  historical  development,  each  being 

■ » 

designed  to  afford  different  options  to  the  experimenter. 


2.4.X  Single-stage  procedure 

The  easiest  type  of  procedure  to  implement  is  a single-stage  one. 

The  following  single-stage  procedure  was  proposed  by  Bechhofer  [1954];  constants 
c^  (see  a),  below)  necessary  to  implement  this  procedure  are  given  in  Table  I, 

"a)  Take  a common  number  N of  observations  from  each  of  the  k 

populations  where  N*  is  the  smallest  integer  greater  than  or 

2 

equal  to  (c^  *0/6*)  . ■ 

N 

b)  Calculate  ^ b ^ X^/N  and  let  X^  < X[2]  <*  “ < X[k] 

3 “i  __  (26) 

denote  the  ranked  values  of  the  X^.  ■ 

. c)  Select  the  population  which  yielded  Xj-^-j  as  the  one  associated 

with  Uruv" 


Note;  The  constants  are  computed  under  the  assumption  that  the 

are  the  so-called  least- favorable  (LFP- configuration,  i.e., 

"Cl]  ■ tk-l]  ■ UCk]  - 


Table  I 


% 

I 

1 

& 

I 


pi 

1 1 


Values  of 

Ck,P* 

. 

k 

2 

3 

4 

5 

7 

3.2900 

3.6173 

3.7970 

3.9196 

4.0861 

2.3262 

2.7101 

2.9162 

3.0552 

3.2417 

1.8124 

2.2302 

2.4516 

2.5997 

2.7972 

1.1902 

1.6524 

,1.8932 

2.0528 

2.2639 

0.3583 

0.8852 

1.1532 

1.3287 

1.5583 

0.99  3.2900  3.6173  3.7970  3.9196  4.0861  4.2456 

0.95  2.3262  2.7101  2.9162  3.0552  3.2417  3.4182 

0.90  1.8124  2.2302  2.4516  2.5997  2.7972  2.9829 

0.80  1.1902  1.6524  ,1.8932  2.0528  2.2639'  2.4600 

0.60  0.3583  0.8852  1.1532  1.3287  1.5583  1.7700 

The  values  in  this  table  are  abstracted  from  Table  I of  Bechhofer 
[1954]  where  values  for  other  k and  P*  are  also  given.  Addi- 
tional values  for  k = n+1  « 2(1)51  and  P * a 0.99,  0.975,  0.95, 
0.90,  0.75  are  contained  in  Table  I of  Gupta  [1963]i  Gupta's  values 
must  be  multiplied  by  in  order  to  obtain  the  piV  - values 
requited  in  (2.6). 
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2.4.2  Open-ended  sequential  procedure  without  elimination 


The  single-stage  procedure  of  Section  2.4.1  is  conservative  in  the 

V 

sense  that  the  constants  pA  necessary  to  implement  it  are  computed  under 


the  assumption  that  the  population  means  are  .in  the' LF-configuration;  however, 
it  has  been  shown  (Hall  [1959])  that  the  probability  requirement  (2.2)  cannot 
be  guaranteed  with  a smaller  N if  the  experimenter  restricts  consideration 
to  single-sta^e  procedures . If  this  restriction  is  eliminated,  and  multistage 
procedures  are  permitted,  then  certain  gains  can  be  achieved.  What  is  desired 
is  a multi-stage  procedure  which  not  only  will  guarantee  the  probability  require- 
ment (2.2)  when  the  population  means  are  in  the  LF-configuration,  but  also  will 
require  a smaller  number  of  observations  per  population,  on  the  average,  than 
the  N of  (2,6)  when  the  population  means  are  in  very  favorable  configurations-- 
in  particular  when  (y^  “ ^[k-l]^0  *s  larSe.’  T^e  following  sequential 
procedure,  which  possesses  these  attributes,  was,  proposed  by  Bechhofer,  Kiefer, 
and  Sob el  [1968],  pp.  258-9,  264-7. 


"a)  Take  one  observation  from  each  of  the  k populations  at 


each  stage  of  experimentation.  Let  ][  X. . denote  the 

j=l  13 

cumulative  sum  from  II.  (l^i^)  at  the  mth  stage  of 

• m m m 

experimentation,  and  let  ][  X^.^  < £ xj-2]j  <tt><  I > 

j — 1 j - 1 j--l 

m 

denote  the  ranked,  values  of  the  £ X. . . 

j = l 13 

b)  At  the  mth  stage  of  experimentation  (m=l,2,...)  compute 


denote  the  ranked,  values  of  the 


k-1 

Z = l exp 
m i»l 


j 1 xr 


_5 * j=i 


Then  proceed  as  fellows: 


i)  If  Z .<  (1-Pft)/P»',  stop  experimentation  and  select  the 
m m 


population  which  yielded  £ Xr.,, 
with  w[k ]‘ 


as  the  one  associated 


t 


ii)  If  Z > (1-P*)/P*,  take  another  observation  fropi  each  of 
m 

the  k populations  and  compute 
Continue  in  this  manner  until  the  rule  balls  for  stopping.” 

Remark  4 ; For  (2.7)  the  observations  are  taken  in  vectors,  each  vector  consti- 
tuting a stage , there  being  one  observation  from  each  population  in  every  vector. 
The  number  of  stages  (i.e.,  number  of  observations  per  population)  necessary  to 
terminate  experimentation  is  a random  variable.  The  expected  number  of  stages  to 


terminate  experimentation  has  been  shown  (B-K-S  [1968],  Tables  12.8.2  and  12.8.3) 
to  be  less  than  N for  many  configurations  of  the  ui  (l<,i<)0;  in  particular, 
if  ^[k]  ’ W[k-lj^°  *s  ,lai,2e*  then  w*th  high  Probability  experimentation  will 
cease  after  only  a small  number  of  stages.  Regardless  of  the  configuration  of 
the  (l£i<)0  experimentation  will  cease  with  probability  one  after  a finite 

number  of  stages. 

2.4.3  Closed  sequential  procedure  with  elimination 

The  sequential  procedure  of  Section  Z.1*.?  has  two  possible  drawbacks: 
i)  It  is  openended,  i.e.,  before  the  start  of  experimentation  it  is  not  possible 
to  give  a finite  upper  bound  on  the  number  of  stages  to  terminate  experimentation, 
and  ii)  It  does  not  eliminate  "non-contending”  populations,  i.e.,  it  continues  to 
sample  from  populations  which,  based  on  observations  obtained  in  the  early  stages 
of  experimentation,  would  appear  to  be  out  of  contention  for  being  selected  as 
"best.”  The  following  sequential  procedure,  which  overcomes  these  drawbacks 
of  (2.7),  was  proposed  by  Paulson  [1964];  like  (2.7)  it  guarantees  the  probability 
requirement  (2.2)  when  the  population  means  are  in  the  LF-configuration , and 
also  tends  to  cease  experimentation  early  when  the  population  means  are  in  very 
favorable  configurations: 

For  fixed  A (0<A<6*V2)  let  = [o2/(6*-A)]log[(k-l)/(l-P*)],  and 
1st  = the  largest  integer  lass  than  a^/A.  Paulson’s  procedure  is  actually 
a family  of  procedures  which  depend  on  the  choice  of  A;  in  Remark  9,  below, 
wa  shall  make  some  comments  on  the  role  of  A. 


"Take  one  observation  from  each »of  the  k populations  at  the 
first  stage  of  experimentation.  Eliminate  from  further  considera- 
tion any  population  It.  for  which  a,  - A < max  X , - X.,.  If 

1 x 1 <s&  01  11 


all  but  one  population  is  eliminated  after  the  first  stage,  stop 
experimentation  and  select  the  remaining  population  as  the  one 
associated  with  1 p^]*  Otherwise,  go  on  to  the  second  stage,  and  take 
one  observation  from  each  population  not  yet  eliminated.  At  stage 
m (2<jn<W^)  take  one  observation  from  each  population  not  eliminated 
after  the  (m-l)st  stage,  and  then  eliminate  from  further  considera- 
tion any  remaining  population  Jl^  for  which 


m 


m 


ml  < max{  l Xsj)  - l X 


j=l 


(2.B) 


where  the  sums  are  only  for  populations  left  after  the  (m-l)st 
stage.  If  all  but  one  population  is  eliminated  after. the  mth 
stage,  stop  experimentation  and  select  the  remaining  population 


as  the  one  associated  with 


^[k]’  otherwise  go  on  to  the  (m+l)st 


stage.  If  more  than  one  population  remains  after  stage  W^, 
terminate  experimentation  at  the  (W^+Dst  stage  by  selecting 
the  remaining  population  with  the  largest  sum  of  the  (W^+l) 
observations  as  the  one  associated  with  p. 


'[k  y 


Remark  5:  The  procedure  (2.8)  never  requires  more  than  W^+l  stages  to  terminate 

experimentation. 

Remark  6;  The  procedure  (2.8)  permanently  eliminates  apparently  non-contending 
populations;  thus  the  number  of  observations  taken  at  the  mth  stage  of  experi- 
mentation is  less  than  or  equal  to  the  number  of  observations  taken  at  the 
(m-l)st  stage  of  experimentation. 


Remark  7;  The  cost  of  experimentation  using  procedures  (2.7)  and  (2.8)  can  be 
measured  in  terms  of  expected  number  of  stages  to  terminate  experimentation 
and/or  expected  total  number  of  observations  to  terminate  experimentation. 
Which  one  is  an  appropriate  measure  will  depend  cn  the  practical  situation  at 
hand . 

Remark  0:  Romberg  [1966]  has  demonstrated  using  Monte  Carlo  sampling  methods 

that 
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and 


max  E{Number  of  stages  to  terminate  experimentation} 

Wj.Wj....,Uk 

1 » 


max  E{Total  number  of  observations  to  terminate  experimentation} 
^1 *y2 1 * * * 


are  less  for  (2.8)  than  for  (2.7)  when  P*  is  high  (i.e.,  close  to  unity)  but 
the  inequality  is  reversed  if  P*  is  sufficiently  small;  Perng  [1969]  has 
studied  that  question  analytically.  This  result  is  of  practical  interest 


since  it  compares  the  performance  of  (2.7)  and  (2.8)  when  y 


[1]  " wtk ]* 


i . e . i 


when,  unknown  to  the  experimenter,  all  of  the  population  means  are  equal  and 
thus  the  expected  number  of  stages  and  the  expected  total  number  of  observations 
are  at  their  maxima. 


Remark  9:  Fabian  [1974]  pointed  out  the  advantage  of  choosing  X = 6*/2,  and 

recommended  for  that  choice  of  X that  l-P*  in  be  replaced  by  2(1-P*') 

yielding  a^2  s [2o2/d*]log[Oc-l)/2(l-P*)]  = b (say);  then  b replaces 
a^  and  6*/2  replaces  X in  (2.8).  This  modified  procedure  still  guarantees 
the  probability  requirement  (2.2)  when  the  population  means  are  in  the 
LF- con figuration.  It  uniformly  (in  the  y^)  reduces  the  expected  number  of 
stages  and  expected  total  number  of  observations  relative  to  the  ones  that 
would  have  been  obtained  with  the  unmodified  procedure  employing  X = 6»'/2; 
in  addition,  in  either  the  family  of  unmodified  Paulson  pi’ocedures  or  in  the 
family  of  modified  Paulson  procedures  the  choice  X * 6*/2  has  the  property  that 
max  E{Total  number  of  observations  to  terminate  experimentation}  is 
^1 *^2 * * * * *^k 

approximately  minimized  for  P*  close  to  unity. 

2.4.4  Two-stage  procedure 

The  sequential  procedures  (2.7)  and  (2.8)  have  the  drawbacks  that 
they  may  not  be  appropriate  for  use  in  certain  types  of  experimentation.  For 
example,  in  agricultural  experimentation  where  yields  can  be  obtained  only 
once ‘p°r  year  (or  per  growing  season),  and  thus  only  one  vector  of  observations 
can  be  obtained  per  time  period,  multi-stage  experimentation  is  impractical. 
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» , . ' -'ll 

w. 

In  auch-  situations  two-stage  experimentation  would  appear  to  be  appropriate. 

Alam  [1970]  and  Tamhane  [197S]  havo  developed  two-stage  procedures  which  guarantee 
the  probability  requirement  (2.2)  when  the  population  means  are  in  the  LF- 
configurationj  their  procedures  screen  out  the  apparently  non-contending  popu- 
lations in  the  first  stage,  and  concentrate  sampling  on  the  remaining  populations 
in  the  second  (terminal)  stage.  . Tamhane 's  procedure  has  the  added  virtue  of 
possessing  a minimax  property  similar  to  that  achieved  by  Fabian’s  modification, 
of  (2.8)  when.  ?.  * 6*/2. 

2 . 5 Procedures  for  use  with  the  indifference-zone  approach  under  the 
assumption  of  common  unknown  or  completely  unknown  variances 

As  was  mentioned  in  Section  2.3.2,  if  the  experimenter  wishes  to 
guarantee  (2.2)  and  adopts  assumption  (2.5e)  or  (2.5d)  then  he  cannot  use  a 
single-stage  procedure.  In  this  section  we  shall  consider  two- stage  procedures 
which  accomplish  these  objectives. 


8 


2 . J>  . 1 Two-stage  procedure  for  the  common  unknown  variance  case 

The  following  two-stage  procedure  for  the  common  unknown  variance 
case  was  proposed  by  Bechhofer,  Dunnett,  and  Sobel  [1954);  constants  h.  nA 

K | r” } n 

(see  c),  below)  necessary  to  implement  this  procedure  for  P*  = 0.95  are 
given  in  Table  II. 

”a)  In  the  first  stage  take  an  arbitrary  common  number'  NQ  > 1 

of  observations  from  each  of  the  k populations. 

9 k N N , 

b)  Calculate  S = £ (X..  - ]>0  X../N.)  /n  which  is  an 

i=l  j=l  13  j=l  13  0 

o 

unbiased  estimate  of  a based  on  n = kCN^-l)  degrees  of 
freedom. 

c)  Enter  the  appropriate  table  (e.g.,  Table  II,  below,  for 

P*  = 0.95)  with  n = k(Ng-l)  and  the  specified  PA,  and 
obtain  a constant  h^  n = h (say). 

d)  In  the  second  stage,  take  a common  number  N-NQ  of  additional 
observations  from  each  of  the  k populations  where 


9^0  ‘ 


1 


N «=  N0  if  ?(hS/S*r  < N0 

H « C20iS/6*)23  if  2(hS/6?)S  > NQ, 


and  Cy]  denotes  the  smallest  integer  equal  to  or  greater  than  y. 

e)  Calculate  the  k over-all  (first-stage  plus  second  stage)  sample 

N N N 

•ums  l X..  dii-^),  and  let  l Xn,.  •<  l Xr  <...< 

jBl  3-3  jBl  LXJ3  jBl  *•*  J j 

N N 

1 Xrj,n  denote  the  ranked  values  of  the  ][  X,.. 

j«l  LKJ3  j=l  13 

N 

f)  Select  the  population  which  yielded  £ Xr.  , . as  the  one 

j=l  lKJJ 

associated  with  m • 11 

Note:  The  constants  h^  pA  n are  computed  under  the  assumption  that  the 

(l<i<)<)  are  in  the  LF-configuration. 


Table  II 


Values  of  h,.  „A  _ for  P*  - 0.95 
' 1 - ■ K>P**>n  1 ■ 


2 

3 

. 4 

* 

5 

7 

10 

5 

2.02 

2.44 

2.68 

2.05 

3.08 

3.30 

6 

1.94 

2.34 

2.56 

2.71 

2.92 

3.12 

7 

1.89 

2.27 

2.48 

2.62 

2.82 

3.01 

8 

1.86 

2.22 

2.42 

2.55 

2.74 

2.92 

9 

1.83 

2.18 

2.37 

2.50 

2.68 

2.86 

10 

1.81 

2.1S 

2.34 

2.47 

2.64 

2.81 

IS 

1.75 

2.07 

2.24 

2.36 

2.51 

2.67 

20 

1.72 

2.03 

2.19 

2.30  ' 

2.46 

2.60 

30 

1.70 

1.99 

2.15 

, 2.25 

2.40 

2.54 

60 

1.67 

1.95 

2.10 

2.21 

2.35 

2.4« 

• 

1. 64 

1.92 

2.06 

2.16 

2.29 

2.42 

. 9d 


The  values  In  this  table  are  abstracted  from  fable  la  of 
Dunnett  [1955];  Table  lb  of  Dunnett  [1955]  gives  corre- 
sponding values  for  P*  « 0.99;  Dqnnett's  p equals 
our  k-1. 

Note:  The  value  of  h.  given  for  the  n ■ row  of  Dunnett  [1955],  Table  Is, 

— k,P",n 

is. the  same  as  the  value  given  by  Gupta  [1963],  Table  I,  for  the  same  k-1  = p * n 
and  P*  » 0.95  = l-o. 

Remark  10:  The  total  number  of  observations  N required  by  the  two-stage  procedure 

. . ' . * “ 2 

is  a random  variable  since  its  value  depends  on  the  value  of  S ; no  additional 

observations  are  taken  in  the  second  stage  if  S is  sufficiently  small. 

Remark  11;  Paulson  [1964],  Section  5,  proposed  an  open-ended  sequential  procedure 
which  permanently  eliminates  non-contending  populations;  hie  procedure  is  appli- 
cable in  situations  in  which  the  common  variance  is  unknown . 

2.5.2  Two-stage  procedures  for  the  complately  unknown  variance  case 

Dudewicz  and  Dalai  [1971],  and  also  Rinott  [1974],  proposed  two- 
stage  procedures  for  the  completely  unknown  variance  case.  Like  (2.9),  the 
common  number  of  observations  in  the  first  stage  for  each  of  these  procedures 
is  arbitrary  (>1)»  while  the  total  number  of  observations  per  population  is 
a random  variable. 

2 . 6 Procedure  for  use  with  the  subset  approach  under  the  assumption  of 
common  (known  or  unknown)  variance 

As  was  mentioned  in  Section  2.3.2,  if  the  experimenter  wishes  to 
guarantee  (2.4)  and  adopts  assumption  2.5a)  or  2.5c),  thon  he  can  use  a single- 
stage  procedure.  The  following  single-stage  procedure  was  proposed  by  Gupta 
[1956],  [1965]  for  use  under  assumption  2.5c);  constants  d^  (see  c),  below) 

necessary  to  implement  thio  procedure  ore  given  in  Table  III.  (Under  assumption 
2.5a,  tha  random  variable  S in  d)  of  (2.10)  is  replaced  by  o,  and  the  value 
of  d^  n for  n » •»  is  used.) 


9^2 


"«)  Tako  a common  arbitrary  number  N > 1 of  observations  from 

aaoh  of  the  k populations.  . , 

N . 

b)  Calculate  ' l X^/N  (l<i<}0  and  Itt  X^  < <...<  5f[k] 

denote  the  ranked  values  of  the  X. ; also- calculate 
2 k N N 2 

e I V (X..  - l X../N)  /n  which  is,  an  unbiased  estimate 
i-1  j*l  ij 


of  o*  based  on  n » k(N-l)  degrees  of  freedom. 


(2.10) 


c)  Enter  the  appropriate  table  (e.g. , Table  III,  below,  for 
P*  * 0.95)  with  n b k(N-l)  and  the  specified  P*,  and 
obtain  a constant  \ Y>tt  n r & (say). 

d)  Retain  the  population  (l<,i<)<)  in  the  selected  subset  if 

and  only  if  Xj,  >5?^  - dS/vfi\" 


Table  III 

Values  of  d,_  _ for  P*  * 0.95 

"*  ' ~r_  K 1 1 **  j n 11  1 


n 

k 

2 

5 

10 

15 

2.48 

. 3.34 

3.78 

20 

2.44 

3.25 

3.67 

30 

2.40 

3.19 

3.59 

60 

2.36 

3.12 

3.50 

The  valuos  in  this  table  are  abstracted  from  Tabic  I of  Gupta 
and  Sobol  [1957]  which  gives  many  additional  d-values  for 
P*  b 0.75,  0.90,  0.95,  0.975,  0.99. 


dk,P*,n  * ^ hk,PA,n  whore  hk,F*,n  ls  r’1Ven  ln  Table  “•  ' 

Remark  12;  The  width  of  the  ••yardstick"  in  d)  of  (2.10)  is  dS/*^lT  which  decreases 
with  N;  thus  the  larger  the  value  of  N,  the  smaller  the  expected  number  of 
populations  that  will  be  included  in  the  selected  subset.  Also,  for  fixed  N 


, 9^3 


the  more  favorable  the  configuration  of  the  population  means  (e .g. , the  larger 
the  value  of  (u^  * ^[k-I]^^*  the  smaller  the  expected  number  of  populations 
that  will  be  included  in  the  selected  subset.  (This  expected  number  always  lies 
between  unity  and  kP* . ) 

Remark  13 i In  practice  the  subset  approach  is  often  used  for  screening  purposes, 
since  it  tends  to  eliminate  "non-contending"  populations  (i.e.,  those  with  small 
p-values)  from  the  selected  subset.  The  populations  retained  in  the  subset  can 
then  be  subjected  to  further  study  in  an  independent  follow-up  experiment  in 
which  the  indifference-zone  approach  (say)  is  used. 

2.7  Factorial  experiments  involving  means 


The  statistical  model  given  in  Section  2.1  is  appropriate  for  single- 

factor  experiments.  In  a two-factor  experiment  we  have  rc  normal  populations  ' 

- _ 2 

Njj  l£j£c)  with  population  means  and  population  variances  o^. 

It  is  sometimes  appropriate  to  assume  that  u,.  * y + a.  + 3.  ( £ a.  = ][  B.  c ( 

1 J i*l  1 jsi  J 

i.e.,  that  there  is  no  Interaction  between  the  factors,  and  that  = o2 

lijio).  Here  the  and  the  B^  are  referred  to  as  the  "effects"  of 

the  first  and  second  factor,  respectively.  It  is  assumed  that  u,  the  a,, 

2 1 
the  Bj,  and  o are  unknown.  Let  a[i]  Sk  a[2]  »’  * •—  a[r]  and 

B[i|j  <,  Bj-2|j  <,...<,  Bj-c-j  denote  the  ranked  values  of  the  and  the  B^;  it 

is  assumed  that  the  pairing  of  the  It^  with  the  e»^  and  Bj-^-j  (l£i£P»  lij^c) 

is  completely  unknown. 

In  the  above  setup  it  is  possible  to  consider  goals  such  as 


Goal i "To  select  the  ’level'  of  the  first  factor  associated 


with  o, 


and  simultaneously  to  select  the  'level1 


of  the  second  factor  associated  with  S|-0-j," 

with  associated  probability  requirements.  Such  problems  are  treated  for  the 
indifference-zone  approach  in  Section  4 of  Bechhofer  [1954],  The  virtue  of 
conducting  factorial  experiments  in  this  situation  is  discussed  by  Baw a [1972]. 
The  indifforonco-zone  selection  p 'ocedures  of  Sections  2.4  and  2.5  can  be  used 
' in  multi-factor  experiment)  it  is  only  necessary  to  make  appropriate  modifica- 
tions in  the  procedures. 


y.  , 

,y 


It  is  also  possible  to  conduct  single-factor  or  multi-factor  ranking 
and  selection  experiments  using  the  standard  experimental  designs  such  as  ran- 
domized blocks  and  Latin  squares , and  these  designs  ,.play  the  same  type  of  role 
here  as  they  do  in  classical  hypothesis-testing  situations. 

2.8  Means  vs.  a fixed  known  standard 


In  Section  2. 4. 1-2. 4. 4 and  2. 5. 1-2. 5. 2 the  selection  procedures 
proposed  were  devised  to  select  the  category  associated  with  the  largest 
p-value.  However,  in  certain  classes  of  experiments  even  the  "best"  one  of 
the  competing  categories,  i.e.,  the  category  with  the  largest  p-value,  may  not 
be  good  enough  to  warrant  the  experimenter’s  selecting  it.  For  example,  if  the 
competing  categories  are  drugs,  the  best  one  may  not  be  worthy  of  consideration 
unless  the  expected  period  of  immunity  obtained  with  that  drug  is  at  least  some 
specified  period  of  time;  or  if  the  competing  categories  are  types  of  heat 
treatment  of  steel,  the  bast  one  may  not  ba  deemed  satisfactory  unless  the 
expected  tensile  strength  resulting  from  that  type  of  treatment  is  at  least 
some  specified  minimum  value.  Such  types  of  nrobloms  involving  comparisons 
of  means  with  a fixed  known  standard  are  considered  by  Bechhofer  and  Turnbull 


[1974 ] , [1975a];  in  the  first  paper  a single-stage  procedure  is  proposed  under 
assumption  (2.5a),  and  in  the  second  a two-stage  procedure  is  proposed  under 
asBumption  (2.5c).  These  procedures  are  generalizations  of  Bechhofer  [1954] 
and  Bechhofer,  Dunnett,  and  Sobel  [1954].  Gupta  and  Sobel  [1958]  proposed  a 
single-stage  procedure  for  this  problem  using  the  subset  approach. 

3.  The  normal  variances  problem 


Section  2 dealt  with  the  normal  means  problem.  Corresponding  procedures 
exist  for  the  normal  variances  problem.  Ranking  and  selection  problems  involving 
variances  arise,  for  example,  when  the  ordnance  engineer  is  interested  in 
selecting  that  type  of  projectile  which  yields  the  smallest  dispersion  of 
range,  on  when  the  laboratory  technician  is  interested  in  selecting  that 
measuring  instrument  which  has  the  highest  precision  (e.^. , that  scale  which 
has  the  greatest  reproducibility ) . An  analogue  of  the  single-stage  procedure 
given  in  Bechhofer  [1954]  for  normal  moans  is  given  in  Bechhofer  and  Sobel 
[1954]  for  normal  variances}  factorial  experiments  involving  variances  are 
.treated  in  Bechhofer  J[196Ba]  and  [1966b]  using  a model  proposed  in  Bechhofer  [I960]. 
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Bechhofer  and  Turnbull  [1975b]  is  the  counterpart  for  variances  of  Bechhofer  and 
Turnbull  [1974].  An  .analogue  of  the  procedure  given  in  Gupta  [1956]  for  normal 
means  ia  given  in  Gupta  and  Sobel  [1962]  for  normal*' variances. 


4.  The  Bernoulli 


Lem,  and  other  problems 


Ranking  and  selection  problems  involving  Bernoulli  p's  (i.e.,  probabilities 
of  "success"  on  a single  trial)  arise,  for  example,  when  a consumer  is  interested 
in  selecting  that  producer  whose  product  has  the  smallest  fraction  defective. 

An  analogue  of  the  procedure  given  in  Bechhofer  [1954]  for  normal  means  is 
given  in  Huyett  and  Sobel  [1957]  for  Bernoulli  p's,  while  the  counterpart  of  the 
procedure  given  in  Gupta  [1956]  for  normal  means  is  given  in  Gupta  and  Sobel 
[1960]  for  Bernoulli  p's. 

Sobel  [1954]  proposed  a sequential  procedure  for  selecting  tho  exponential 
population  with  tho  largest  mean;  his  results  have  applicability  in  reliability 
studios.  Bechhofer,  Kiefer,  and  Sobel  [1968],  p.  63  considered  sequential 
procedures  for  ranking  parameters  of .certain  stochastic  processes  such  as  the 
Poisson  process  and  the  Wiener  process.  Various  research  workers  have  proposed 
procedures  for  many  other  ranking  and  selection  problems  involving  parameters 
of  distributions  arising  in  practice. 


5.  Closing  remarks 

The  ranking  and  selection  formulation  of  statistical  problems  involving 
inferences  concerning  k > 2 categories  has  wide  applicability  in  the  solution 

v 

of  problems  arising  in  experimentation.  In  this  paper  we  have  sketched  only 
a small  number  of  the  relevant  ideas  and  procedures.  The  interested  reader 
is  raferred  to  Bechhofer,  Kiefor,  and  Sobel  [1960]  for  references  up  to  that 
date,  and  to  Gupta  and  Panchapakesan  [1972]  for  references  to  the  latter  date 
concerning  the  subset  approach.  Additional  and  more  recent  references  are  given 
by  Wotheri.il  and  Ofosu  [1974].  The  writer  would  appreciate  learning  of  experi- 
mental situations  in  which  some  of  the  procedures  described  herein  have  proved 
helpful. 
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